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Abstract. Although there is no universal agreement that students should learn programming, 
many countries have reached a consensus on the need to expose K-12 students to Computational 
Thinking (CT). When, what and how to teach CT in schools are open questions and we attempt to 
address them by examining how well students around the world solved problems in recent Bebras 
challenges. We collected and analyzed performance data on Bebras tasks from 115,400 students 
in grades 3-12 in seven countries. Our study provides further insight into a range of questions ad¬ 
dressed in smaller-scale inquiries, in particular about the possible impact of schools systems and 
gender on students’ success rate. 

In addition to analyzing performance data of a large population, we have classified the con¬ 
sidered tasks in terms of CT categories, which should account for the learning implications of the 
challenge. Algorithms and data representation dominate the challenge, accounting for 75-90% of 
the tasks, while other categories such as abstraction, parallelization and problem decomposition 
are sometimes represented by one or two questions at various age groups. This classification can 
be a starting point for using online Bebras tasks to support the effective learning of CT concepts 
in the classroom. 

Keywords: computational thinking, Bebras challenge, task difficulty, gender issues. 


1. Introduction 

Regardless of the variety of approaches, there seems to be “a convergence towards com¬ 
putational thinking as a core idea of the K-12 curricula” (Hubwieser et al 2014). The 
term computational thinking (CT) was coined by Papert (Papert, 1980) but gained in¬ 
creased attention beginning in the 2000s after Wing published her seminal article posi- 
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tioning CT as a “universally applicable attitude and skill set everyone, not just computer 
scientists, would be eager to learn and use” (Wing, 2006). Since then, many countries 
have introduced in their school curricula CT or other computer science topics, includ¬ 
ing programming (McCartney and Tenenberg, Eds., 2014, Barendsen and Steenvoorden, 
2016, Sentance et al, 2013, Tort and Drot-Delange, 2013, Barendsen et al , 2016, Heintz 
et al ., 2016). 

CT is meant to encompass a set of concepts and thought processes that aid in ap¬ 
proaching problems arising in different fields in a way that could involve computers 
(Mannila et al , 2014). However, the concepts central to CT can be quite foreign to 
many K-12 teachers. Thus, among other challenges, it is important to get both teach¬ 
ers and students engaged by developing learning materials that can provide a positive 
first experience. A compelling example of this kind of material is provided by the Be- 
bras community, which annually arranges an international problem-solving challenge 
aimed at children and youth aged 8-18 (Dagiene and Sentance, 2016, Bezakova and 
Winczer, 2011). The challenge includes motivating and age-adapted tasks relating to, 
for instance, programming and algorithms, computational thinking and information 
technology. Moreover, it does not require any background in these areas. All material 
from past challenges are available online freely for anyone to use, and each task in¬ 
cludes specific information not only on the correct answer, but also on the connection 
to computer science topics. 

A working group at the ITiCSE 2015 conference started an exploratory investiga¬ 
tion of the “task based assessment” approach of the Bebras challenge within the broader 
framework of K-9 computer science education, which encompassed the development 
of national curricula and guidelines as well as teachers’ attitudes toward their practice 
in this field (Barendsen et al , 2015). In particular, their final report proposed a clas¬ 
sification of the conceptual content and question structure of Bebras tasks from years 
2010-2014 according to CT categories. Algorithms and data representation were found 
to be the main concept categories in the K-9 context. Moreover, a taxonomy for ques¬ 
tions related to algorithms was outlined. 

In this study, we have analyzed in terms of concept coverage, presumed 1 and actual 
performance a selection of Bebras tasks for seven countries based on data collected in 
2014 and 2015. Given that many countries are in the process of updating their curricula, 
it is likely that the Bebras challenge was one of the first opportunities for students in 
those countries to be exposed to CT. 

The study extends previous work on Bebras tasks (see subsection 2.2) by answering 
the following research questions based on large multinational cohorts: 

• Which CT concepts and skills are addressed by Bebras tasks? 

• Is estimated task difficulty related to the CT conceptual content? how well does 
the estimated difficulty match the observed students’ performance? 

• Is gender a significant factor in performance for any country or age group? 

The first question is important in order to explore the implications of CT conceptual 
content both in terms of curricular coverage and cognitive demand. The aim of the sec- 


According to the difficulty level assigned by the task proponents. 
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ond question is to identify whether some concepts and tasks appear to be more difficult 
for student and hence might need more scaffolding when introduced at school. Finally, 
the third question is important in order to contribute to the knowledge and discussion on 
raising girls’ interest in computer science. All in all, addressing these research questions 
will reveal to what extent Bebras tasks match the need for suitable CT teaching material 
in K-12 education. 

By analyzing performance data of a large population and from rather different con¬ 
texts, our study provides further insight into a range of questions recently addressed in 
smaller-scale inquiries, in particular about the possible impact of schools systems and 
gender on learning CT concepts. Distinctive characters of our present work are indeed 
the cohort size, amounting to 115,400 students in grades 3-12, as well as the variety of 
educational contexts in the seven countries that have been considered. In addition, the 
paper represents one of the first attempts to explore the implications of the overall con¬ 
ceptual content of Bebras tasks within a CT-related framework. 

The rest of the paper is organized as follows: we first introduce the Bebras challenge 
and review previous work studies relevant for the present analysis. Then we describe the 
set of data collected, and analyze them both qualitatively and quantitatively in terms the 
concept coverage, performance and difficulty level of a selection of the tasks. Finally, we 
end with some conclusions and recommendations. 


2. Background 


2.1. The Bebras Challenge 

The Bebras challenge 2 is an online competition that was introduced in 2004 in Lithuania 
as an outreach instrument, to encourage interest and motivation in computing-related 
topics (Dagiene, 2006). Since then, it has spread worldwide reaching 1.3 million of pu¬ 
pils from 38 countries in the autumn 2015’s challenge and more than 1.5 million from 
30 countries in the 2016’s challenge. 

Bebras tasks are short and answerable in a few minutes through a computerized in¬ 
terface. The task set is created on a yearly basis through crowd-sourcing from participat¬ 
ing countries, which are then further reviewed and polished during an annual workshop 
(Dagiene and Stupuriene, 2016). Most of the tasks we have analyzed are either multi¬ 
choice or interactive. Multi-choice tasks have four nontrivial and well-defined answer 
choices with only one correct solution. Interactive tasks provide a scene or diagram on 
the screen that participants can interact with to carry out actions and change the scene’s 
state to move towards the question’s goal; this interaction may provide feedback on the 
status of the task. 

The challenge is arranged annually in their local languages throughout the world. 
Most countries run their event on Bebras Week , which falls on the second full week of 
November (southern hemisphere countries may use a different week). 


http://www.bebras.org 
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Participants are divided into age groups: Mini (grades 3-4); Benjamin (grades 5-6); 
Cadet (grades 7-8); Junior (grades 9-10); Senior (grades 11-12). The grades corre¬ 
sponding to the age groups may differ slightly between countries based on their local 
curricula 3 . Each age group attempts 15 (9 for Mini) to 18 tasks within 45-55 minutes. 
They will have the same number of tasks, usually 5 or 6 tasks, at each difficulty level: A 
(easy), B (medium) and C (hard) as prescribed by developers. The participants are usu¬ 
ally supervised by teachers who may integrate the challenge in their teaching activities. 


2.2. Related Work 


The popularity of the Bebras’ initiative provides the opportunity to evaluate its results 
in a variety of contexts; this also prompted discussions about the possible implications 
for K-12 computer science education. There is already quite an extensive review of 
work related to the Bebras challenge (Dagiene and Stupuriene, 2016). Therefore, in this 
subsection we will focus on the contributions addressing tasks’ scope and difficulty. We 
also discuss previous work done on CT classification since such work is relevant to the 
qualitative analysis of the Bebras tasks discussed later in the paper. 

Besides their outreach potential (Haberman et al ., 2011, Bell et al , 2011), Bebras 
tasks can be incorporated in the curriculum (Dagiene and Sentance, 2016) or used as an 
instrument to support teachers’ training in computing subjects (Bezakova and Winczer, 
2011), as well as to engage students with CT in connection with their informal training 
for the contest. Gujberova and Kalas present an example of this perspective in order 
to introduce “ modern Informatics as opposed to ICT classes” in K-12 education (Guj¬ 
berova and Kalas, 2013). It is especially from this standpoint that the Bebras challenge 
is an interesting topic of study. 

As discussed in (Kalelioglu et al , 2016), there are a wide range of papers in recent 
years that cover the definitions and classification of Computational Thinking. A key 
source of classification for CT problems is the Computer Science Teachers Association 4 
(CSTA), which has published in cooperation with the ISTE 5 a “Computational Thinking 
Across the Curriculum” table introducing their list of CT categories. In fact, we followed 
the CT characterization proposed in previous work (Barendsen et al , 2015), which is 
inspired by these CSTA/ISTE categories: data collection , data analysis , data represen¬ 
tation , problem decomposition , abstraction , algorithms & procedures , automation , par¬ 
allelization and simulation. The detailed definitions and operational characterization that 
were used for the qualitative analysis are provided in Table 1. 

Bebras tasks are becoming a popular tool to assess computer science competences at 
various levels of school instruction (Duncan and Bell, 2015), to the point that Hubwi- 
eser and Miihling are envisaging a possible future role in PISA-like assessment of such 


3 In particular, South Africa uses also a different labeling of age groups, which we mapped as follows in 
order to have a one-to-one correspondence: Junior (grade 7) —> Benjamin, Intermediate (8-9) —► Cadet, 
Senior (10-11) —► Junior and Elite (12) —► Senior. Other countries not considered in this study have opted 
for different arrangements and/or larger sets of tasks. 

4 http://www.csteachers.org 

5 https://www.iste.org 
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Table 1 

CT definitions from previous work (Barendsen et al., 2015) 


CT category Definition 


Operational definition 


Data collection 
Data analysis 


Data 

representation 


Problem 

decomposition 

Abstraction 


Algorithms & 
procedures 


Automation 


Parallelization 


Simulation 


The process of gathering app¬ 
ropriate information. 

Making sense of data, fin¬ 
ding patterns, and drawing 
conclusions. 

Depicting and organizing data 
in appropriate graphs, charts, 
words, or images. 

Breaking down tasks into 
smaller, manageable parts. 
Reducing complexity to de¬ 
fine main idea. 


Series of ordered steps taken 
to solve a problem or achieve 
some end. 


Having computers or ma¬ 
chines do repetitive or tedious 
tasks. 

Organize resources to simul¬ 
taneously carry out tasks to 
reach a common goal. 
Representation of a model 
of a process. Simulation also 
involves running experiments 
using models. 


Find a data source for a problem area. 

Take data and transform it to solve a problem. Often there 
is some statistical analysis involved in the transformation, 
although the statistics do not have to be sophisticated. 

Take data and put it into a specified format. Includes 
descriptions of data that involve particular structures. It 
may involve understanding the implications of graphs or 
other representations on the solution of a problem. 

Breaking a problem or task into smaller pieces to enable an 
easier or better solution. 

Problems that ask for the creation of a formula. The 
distillation of broader ideas out of narrower concepts. 
Finding rules that apply to a given problem. Finding a 
pattern to model some behavior. Identifying essential facts 
about a structure or problem to verify correct answers. 
Solving maximization, minimization, or other optimization 
problems. Following a step-by-step procedure. Verifying 
potential solutions as valid or invalid. Encoding or 
encryption/decryption problems, including the application 
of an encryption scheme to a sample set of data. Debugging 
solutions and finding errors in a solution. Applying a set of 
rules to determine specific values. Choosing or verifying 
pseudocode or code. 

No instances found. 


Scheduling problems. 


Tasks that are interactive and involved building and 
exploring a solution. 


competences (Hubwieser and Miihling, 2014). However, multiple studies have identified 
the apparent discrepancy between a task’s estimated and perceived difficulty as a major 
concern. Yagunova and colleagues found a correlation between length of statement and 
difficulty for primary school students, and proposed using an “a posteriori task difficulty 
to compensate for differences” (Yagunova et al , 2015). Other researchers investigated 
the psychometric factors which influence success (Hubwieser and Miihling, 2015), ex¬ 
plored how to reliably predict task difficulty (van der Vegt, 2013), and developed statisti¬ 
cal models for this purpose (Bellettini et al ., 2015). 

By exploring the performance of three European countries, Dagiene et al (Dagiene 
et al , 2014) found that the ability to solve tasks was largely independent of the school 
system, and that often supposedly easy tasks turned out to be harder than expected to the 
lower age categories. 
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Bebras performance has also been analyzed from a gender perspective, but the find¬ 
ings appear to be still inconclusive. On the one hand, Dagiene and colleagues were not 
able to uncover any significant difference in overall performance (Dagiene et al ., 2014), 
although a study of the 2014 Lithuanian contest showed that boys outperformed girls 
in specific tasks requiring spatial reasoning and were more likely to guess an uncertain 
answer than girls (Dagiene et al , 2015). On the other hand, an investigation of a large 
cohort taking part in the 2014 German challenge revealed that boys of all age groups 
performed appreciably better compared to girls, the gap increasing strikingly for the 
eldest students (Hubwieser et al ., 2016). 


3. Methodology 


3.1. Data Collection 


Bebras contest data are stored on local servers and initially we had access to anonymized 
data from four countries (Italy, Australia, Finland, and Lithuania); we also received re¬ 
sults (percentage of correct answers per age group and gender) from South Africa, Swit¬ 
zerland and Canada. 

The list of countries and their participation numbers are listed in Table 2. Finland 
(FIN), Lithuania (LIT) and Canada (CAN) provided data for both years. Australia 
(AUS) and South Africa (SAf) ran their challenges in early 2015, but they are listed in 
2014 as they used 2014 tasks. Switzerland (SW) and Italy (ITA) provided data for their 


Table 2 

Participation data for 2014 and 2015, covering 115,400 students 


2014 

AUS 

FIN 

LIT 

SAf 

CAN 

Total 

Mini 

1837 

585 

2409 

- 

- 

9,589 

Benjamin 

2350 

675 

7167 

1558 

- 

10,571 

Cadet 

2568 

1322 

5988 

3059 

- 

13,214 

Junior 

1344 

2140 

6265 

2602 

4558 

13,791 

Senior 

526 

128 

3147 

999 

- 

4,062 


8625 

4850 

24976 

8218 

4558 

51,227 


2015 

ITA 

FIN 

LIT 

SW 

CAN 

Total 

Mini 

1648 

702 

2374 

1026 

- 

10,486 

Benjamin 

904 

2527 

7110 

1947 

- 

11,188 

Cadet 

1622 

1425 

5810 

4905 

4638 

18,704 

Junior 

2083 

3735 

6114 

4124 

5713 

18,959 

Senior 

287 

1207 

3304 

1411 

- 

5,279 


6544 

9596 

24712 

13413 

9908 

64,173 
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2015 challenges 6 . Thus, we have results from a total of 115,400 students from seven 
different countries. 

There is a large Bebras task pool for each given year, out of which each country se¬ 
lects their own local subset (usually 15 questions, 18 for Lithuania) for each age group. 
Some countries use smaller sets, 9 to 12 questions, for the Mini age-group. The collec¬ 
tion of Bebras tasks used in 2014-15 by the countries listed in Table 2 had 145 different 
tasks, of which 88 were used by 2 or more countries. 

3.2. Qualitative Analysis 

In order to address the research questions presented in section 1, we needed to look at the 
data from both a qualitative and a quantitative perspective. As mentioned above (subsec¬ 
tion 2.2), we based our CT classification on a framework developed in the previous work 
(Barendsen et al., 2015), summarized in Table 1. 

The qualitative study was done through content analysis , see e.g. (Cohen et al ., 2013, 
Gavora, 2015), in which either two or three of the authors independently went through 
all selected tasks and assigned each task one or more of the CT categories summarized in 
Table 1. The authors agreed on the majority of assigned categories; in cases of disagree¬ 
ment, one of the authors gave a suggestion for how to resolve the situation, which could 
then be accepted or countered with another suggestion by the other authors. 

After three revisions and related discussion about the most appropriate characteriza¬ 
tion of the tasks assigned in 2014 and 2015 (in the considered countries), an inter-rater 
agreement of 90.2% was attained. A final decision was made by one of the co-authors 
chosen by the others to review and arbitrate disagreements between the rest. None of the 
resulting classifications were disputed by the group as a whole. We may also notice that 
our classification is not always in agreement with the proponents’ intentions, as attested by 
the section “it s informatics ” in the task documentation 7 . More specifically, full agreement 
between both raters and proponents is attained for about 59% of the tasks, whereas there is 
a complete mismatch in 4% of the cases and some substantial difference in further 7%. 

Next, we provide specific examples of the classification to illustrate both the process 
and the way that the CT categories were instantiated in the Bebras tasks. 

Exercise 2015-MY-01 8 shows a broken bracelet made out of six colored beads with 
different shapes as shown in Fig. 1. The student must pick the corresponding bracelet out 
of the four given circular ones. 

This task was classified as data representation (DR). This is because the key to solv¬ 
ing the task is to realize that the circular representation of each of the solutions imposes 
conditions regarding which shapes can be located next to each other. Once the impor¬ 
tance of adjacency of shapes is recognized, it is straightforward to verify which option 
is the correct one. 


6 Italy ran the challenge in teams of 3-4, but the table reects the total number of students. 

7 Actually, the same task may be categorized differently, or with a different emphasis on its informatics/CT 
features, in the documentation provided by organizers from different countries. 

8 Each task is identified by three data: year, proponent country and progressive number. 
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Fig. 1. Task 2015-MY-01. 


Exercise 2015-LT-03 describes an automated system in a chocolate factory. The 
commands that can be given to the decorating robot arm are: Leaf - draws a long oval, 
Circle - draws a circle, Rotate k - rotates the sweet clockwise by k degrees, and Repeat 
n [...] - repeats the commands in the brackets n times. 

An example is given of a sequence of commands and the drawing it would produce. 
Then the drawing in Fig. 2 is provided and the question asks which of the following 
command sequences does not cause the robot to draw the indicated flower. The op¬ 
tions are: 

A. Repeat 6 [Rotate 30 Circle Rotate 30 Leaf]. 

B. Repeat 6 [Leaf Rotate 60] Rotate 330 Repeat 6 [Circle Rotate 300]. 

C. Repeat 6 [Leaf Rotate 60] Repeat 6 [Circle Rotate 60]. 

D. Repeat 3 [Rotate 120 Repeat 2 [Leaf Rotate 30 Circle Rotate 150]]. 

This task was classified as algorithms & procedures (ALG). The problem has pro¬ 
vided various functions defined for the robot and asks which sequence of function calls 
will produce the correct drawing. While the use of procedures might suggest abstraction, 
the exercise does not call for the creation of any procedures. Instead, the problem is 
asking for interpretation and/or verification of a sequence of commands, something we 
classified as algorithms. 

Exercise 2015-CZ-01 provides an example of a Bebras task that was classified into 
multiple CT categories. The exercise asks students to match the “walnut animals” in 
the two rows of Fig. 3 - i.e., to recognize graph isomorphism. This task was classified 
both as data representation (DR) and as abstraction (ABS), the latter because in order 
to match isomorphic graphs students are required to think of their structure at a more 
abstract level. 



oft 


Fig. 2. Task 2015-LT-03. 
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Fig. 3. Task 2015-CZ-01. 

Further examples are the tasks 2015-DE-07, asking for a liquid-pouring plan using 
three different beakers to attain a fair share of a chemical, and 2015-HU-05, essentially 
a towers of Hanoi puzzle of height three. Both tasks are clearly in the ALG class. How¬ 
ever, the former was also classified as simulation (SIM), as the question is interactive so 
that student can try any number of plans before validating the solution. The latter was 
assigned the additional problem decomposition (PD) category, since solving the puzzle 
requires to figure out how freeing the largest items as well as the destination of the tower 
as unavoidable subproblems. 

3.3. Quantitative Analysis 

The quantitative analysis aims to gain further insight into the factors that impact on task 
performance. Firstly, we looked at each country’s performance as a whole and saw if the 
estimated difficulty matches the performance. We also looked at other factors that are 
related to task performance such as age, gender country and type of task. In particular, 
we reconsidered the impact of gender in performance to see if our multinational cohorts 
exhibit the same trends identified in previous studies. In order to accomplish these goals, 
we exploited standard statistics tools (average/mean, clustering and correlation) as per 
data shown in subsection 4.2. 


4. Results 


4.1. Results of Qualitative Analysis 

We now consider the results of the classification into CT categories. 77% of the tasks 
belong to the algorithms & procedures (ALG) category and 42% to data representa¬ 
tion (DR), with 25% of the tasks covering both categories (note only 6% of the dataset 
does not match either ALG or DR). About one fourth of the tasks, on the other hand, 
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are mapped to at least one of the remaining categories reported in Table 1, including 
in addition digital literacy , with a slight predominance of abstraction (ABS: 8%) and 
simulation (SIM: 7%) due to the usage of interactive tasks. In other words, the Bebras 
challenge is mainly focused on algorithmic thinking and data representation. 

A finer-grained picture of the distribution of CT categories over age groups as well 
as over individual challenges is depicted in the Fig. 4 and Fig. 5, which also include the 
concepts data analysis (DA), problem decomposition (PD) and parallelization (PAR); 
note there were no instances of data collection (DC) or automation (AUT). From Fig. 4 


90% 



Fig. 4. Percentage of the CT categories ALG, DR, ABS, SIM, DA, PD and PAR assigned 
in the contests over age groups. 


Australia 2015 (using 2014 tasks) Lithuania 2014 

■ ALG ■ ALG+DR ■ DR * DR+ABS ■ DA □ PD ■ALG ■ ALG+DR ■ DR ^ DA □ PD * DR+DA H ALG+ABS 



mini benjamin cadet junior senior mini benjamin cadet junior 


Italy 2015 Lithuania 2015 

■ALG • ALG+DR ■ DR ^ALG+DA DPAR ^ ALG+PD H ALG+PAR ^ DR+ABS "ALG ■ ALG+DR ■ DR ■ DA ■ PAR □ LIT DABS * DR+ABS «ALG+Other 



mini benjamin cadet junior senior mini benjamin cadet junior senior 


Fig. 5. Percentage of the CT categories as seen by students in 4 of the 7 country challenges. 
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we can see that the percentage of tasks classified as ALG varies between 67% (Mini 
and Senior) and 81% (Junior), whereas the percentage relative to DR falls in the range 
45%-53%, achieving the maximum for Senior. It may also be interesting to observe that 
the largest percentage of tasks implying the ABS category (8%) can be found in the Mini 
or Benjamin challenges, most often together with DR. 

The plots in Fig. 5, on the other hand, show what is the distribution seen by students 
in each age group taking a contest in four of the sample countries 9 . Overall, ALG tasks 
(either alone or combined with other categories) represent at least 2/3 of the tasks as¬ 
signed for each age-group in any country, ranging from about 67% (Switzerland 2015) 
to more than 93% (Canada 2015). The incidence of different CT categories appears to 
depend much more on the year of the task dataset than on the country. As observed in 
Fig. 5, the percentage of tasks classified DR (including ALG+DR) dropped by nearly 
half from 2014 to 2015, replaced by more ALG tasks and with additional but limited 
exposure to tasks from other categories, i.e. ABS, PD and PAR. 

Algorithmic tasks. Algorithmic thinking encompasses a diverse range of algorithms, 
and as it covers 2/3 of the Bebras tasks, it may be interesting to describe in more detail the 
range of questions that belongs to this category. Analyzing the 474 ALG items used in the 
42 considered challenges 10 , we observed the following subtypes and distributions: 

• 9% of the questions require to understand or produce short sequences of com¬ 
mands to achieve some goal; as it may be expected, the bulk of these questions 
(82%) were addressed to the Mini and Benjamin age groups. 

• 18% of the tasks ask students to devise less straightforward (program) strategies 
or plans, such as those needed to carry out a dichotomic search, to instruct a 
“turtle” to draw a shape, or to solve a tower-of-Hanoi puzzle. 

• 12% are either about mastering or combining higher-level (e.g. procedural) be¬ 
havior patterns in a process, 83% of which were addressed to the Cadet, Junior 
and Senior age groups. 

• 26% of them require to carry out some combinatorial analysis and/or to figure out 
strategies to reduce complexity. Half of them have also been classified as DR: typi¬ 
cal such instances, e.g., ask for an optimal path in a graph according to some given 
criteria. 81% of these tasks were again addressed to the last three age groups. 

• Other kinds of task (23% of the items overall) include understanding how data are 
coded, tracking the execution of a program based on some form of evolving state, 
carrying out accurate deductions. 


4.2. Results of Quantitative Analysis 

Task difficulty versus success rate. Each age group attempts three sets of (5 or 6) tasks, 
ranked in increasing difficulty as A (easy), B (medium) and C (hard). We have consid¬ 
ered the following mapping between a priori rank and success rate: A means around 3/4 


9 The others are not shown due to space limitation. 

10 Here the same task is counted as many times as there are contests which include it. 
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of the class will solve the task correctly, B means 1/2 of the class will solve the task, and 
C means only 1/4 of the class will solve the task. 

Fig. 6 shows the success rate versus the expected range of success for the easy, 
medium and hard questions for each challenge. Ideally, all results should fit inside the 
two trend lines plotted in the chart. Points above reflect questions easier than estimated; 
points below reflect harder questions. Overall, Fig. 6 seems to indicate that each con¬ 
test has a reasonable mix of questions, although the rank given to several tasks does not 
match the observed performance. A detailed analysis of the difficulty of each contest 
is provided in Table 3. Ideally, we should have 33% of questions in each actual easy/ 


-LIT- 14 “ UT-15 + ITA + SWISS ° AUS ♦ S.Africa * FIN-14 A FIN-15 A CAN 


V) 

i- 

Q ) 


5 


90% 

75% 

60% 

45% 

30% 

15% 

0 % 



0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 

Expected performance 

A - 75% B - 50% C - 25% 


Fig. 6. Expected versus real success rate for A/B/C tasks in each dataset. 


Table 3 

Spread of a-posteriori easy/medium/hard tasks for each challenge (aiming for 33% each), 
tasks estimated at the right level, and average success rate per country 


^^^Level 

Set 

easy 

>0.65 

medium 

0.65-0.35 

hard 

<0.35 

Rank 

match 

Avge 

success 

SW 

36% 

39% 

25% 

52% 

55.8% 

ITA 

23% 

32% 

45% 

65% 

41.2% 

FIN-14 

23% 

49% 

28% 

45% 

47.3% 

FIN-15 

26% 

40% 

34% 

49% 

49.2% 

LIT-14 

19% 

51% 

30% 

53% 

45.1% 

LIT-15 

15% 

31% 

54% 

40% 

38.6% 

CAN 

38% 

51% 

11% 

53% 

57.4% 

AUS 

28% 

29% 

43% 

72% 

45.6% 

SAf 

15% 

38% 

47% 

68% 

40.2% 
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medium/hard rank. The Lithuanian challenge in 2015 appears to be the hardest. Of the 
84 tasks used among the five age groups, 36 were harder than estimated and 14 were 
easier, with only 34 tagged at the right difficulty level. On the other hand, the Australian 
has a more balanced distribution with only 16 harder and 5 easier tasks out of the 75 
tasks used. 

Success rates by country. The top histogram in Fig. 7 show the average success rates 
for the common tasks shared between the different challenges using the 2014 tasks. The 
point of this graph is not to compare countries, but to put in context the numbers given 
in Table 3. The bottom histogram shows the detail for the cadet 2014 contest, which has 
12 common tasks for the 4 countries. As seen by that plots, when we look at the Cadet 
group, only a few tasks such as 2014-JP-03 (easy) or 2014-DE-05 (hard) can be con¬ 
sidered at the same level for all four countries, while other tasks such as 2014-FR-03 
or 2014-JP-06 are hard for some countries and medium or easy for others. Lithuania, 
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Fig. 7. Average success rate by age groups (top) and task success rate for the 
Cadet group (bottom) for the list of common 2014 tasks. 
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for some particular reason seems to perform the best in some problems while doing 
poorly in others. This variability is also observed in other age groups, both in the 2014 
and 2015 challenges. 

Task performance by task types. The average performance by CT category for each 
age group varied between 40% and 50% with few exceptions. In particular, the success 
rate in ALG tasks ranged from 42% (Mini and Junior) to 45% (Cadet); in DR tasks from 
42% (Mini) to 50% (Cadet). Overall, DR and DR+ALG tasks turned out to be easier 
than purely ALG tasks. The only exception arises with the Mini age group, who showed 
significantly lower performance (37%) in the problems assigned to both the ALG and 
DR classes, probably due to the cognitive load of mastering procedures and data at the 
same time. We also observed low performance for questions requiring data analysis or 
simulation, but the number of tasks was very small. 

By considering the structure of typical ALG tasks outlined in subsection 4.1, the 
tasks requiring to think of a process as a combination of “behavior patterns” turned 
out to be the hardest ones, with a success rate around 31% in average. In particular, an 
extremely difficult instance was 2015-CH-09. This problem described four simple local 
rules to navigate a maze and asked which one of them allowed to escape a given maze. It 
was assigned in Italy and resulted in 2.2% (Cadet), 2.3% (Junior) and 4.9% (Senior) suc¬ 
cess rate. Also the questions asking to detail a non trivial plan to achieve a goal appeared 
to be rather challenging (36% average success rate), especially for the three younger age 
groups. One such example is 2014-FR-03, whose description can be found in subsection 
5.3. On the other hand, perhaps not surprisingly, the easiest questions are those asking 
for short sequences of commands to achieve a goal (56% average success rate), that were 
mostly proposed to Mini and Benjamin students. 

Gender participation and impact on success rate. All countries show similar patterns 
on gender participation, with a balanced participation at lower levels, Mini to Cadet age 
groups, but showing higher participation of boys both in the Junior and in the Senior 
level, as shown in Table 4 11 . 


Table 4 

Gender participation data per country 


Set 

Mini 


Benjamin 

Cadet 

Junior 

Senior 

%F 

%M 

%F 

%M 

%F 

%M 

%F 

%M 

%F 

%M 

SW 

45 

55 

46 

55 

49 

51 

50 

50 

36 

63 

ITA 

45 

55 

44 

56 

44 

56 

23 

77 

21 

79 

FIN-14 

52 

47 

45 

55 

48 

52 

34 

66 

10 

90 

FIN-15 

51 

49 

51 

49 

46 

53 

41 

59 

30 

70 

LIT-14 

43 

56 

46 

54 

44 

56 

42 

58 

30 

70 

LIT-15 

45 

55 

47 

53 

44 

56 

43 

57 

31 

69 

CAN 

- 

- 

- 

- 

43 

52 

38 

62 

- 

- 

AUS 

48 

40 

48 

35 

43 

40 

26 

55 

15 

70 

SAf 

- 

- 

51 

46 

44 

53 

26 

71 

29 

68 


If total is less than 100%, there was a “not reported” option. 
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Regarding the impact of gender on success rate, the correlation of performance pat¬ 
terns (% success) of girls vs. boys is high, ranging from 0.935 (SAf) to 0.98 (LIT). 
This means that task difficulty levels are essentially the same for both female and male 
students. Fig. 8 plots the success rate of female versus male students for each question 
(dashed line in the plot corresponds to boys being twice as successful as girls). Most 
data points are very close to the diagonal, witnessing similar performance regardless 
of gender. However, there are more outliers above than below the diagonal, indicating 
a minor advantage for boys in some tasks, a selection of which we will explore later. 
This feature is almost unnoticeable for the Mini, Benjamin and Cadet age groups (red 
points in the plot), becomes a little more evident for Junior (blue), and emerges clearly 
for Senior (green). 

Interestingly, by drawing linear interpolations relative to each country and age group 
(not shown in the picture), we would notice that on the hardest tasks boys’ performance 
tends to be just slightly better than that of girls - whereas the opposite happens for the 
easiest questions relative to the Mini-Cadet age groups. This observation appears to be 
consistent for all countries. 

By comparing the percentages of successful tasks of girls vs. boys, there is a clear 
trend of worsening performance of girls relative to boys from Mini to Senior age. At 
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Fig. 8. Task success rate correlation of girls (horizontal axis) vs. boys, 
with sample outliers labeled with their relative task code. 
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Senior level boys significantly outperform girls in all countries, the difference in average 
success rate varying from 3.0% (LIT 2015) to 15.0% (SAf). We observed gender varia¬ 
tions by country as follows: 

• The average success rate is systematically in favor of boys in Canada (from 5.5% 
to 7.3%), South Africa (from 0.2% to 15%) and Switzerland (from 0.5% to 6.9%), 
whereas in Australia (from -2.3% to 7.7%) the only exception are Cadet. 

• The results are more balanced in Finland and Lithuania, where girls perform bet¬ 
ter than boys in several contests. Apart from Senior, the difference of the average 
percentages of successful tasks is usually rather small in these countries, but it 
should be noticed that also a deviation of about 1% can be statistically significant 
for a large number of participants. For example, this is the case in favor of Cadet 
boys for LIT 2015: based on the outcome of Welch’s t-test, the difference (1.0%) 
is statistically significant with p-value = 0.020. 


5. Discussion 

5.1. CT Categories 

The categorization outlined in subsection 4.1 covers unevenly the range of CT concepts 
in the chosen framework. At first this makes sense in that both algorithms and data 
representation are core concepts at introductory level. Moreover, it is not easy to figure 
out tasks addressing data collection or automation while meeting the strict constraints 
of the challenge. From a pedagogical perspective, however, we should ask what may be 
an “ideal coverage” of CT concepts and a balanced approach as to the other key con¬ 
cepts, in particular abstraction and problem decomposition. Abstraction, for instance, 
is usually considered more advanced and we may expect, on the one hand, that only a 
small percentage of tasks involve it. On the other hand, it is surprising that the majority 
of such tasks can be found in the challenges addressed to the youngest age groups, to 
whom abstraction should be especially hard. So, perhaps there is room for designing 
more tasks implying abstraction - or problem decomposition - at an appropriate level 
of difficulty. 

Our previous work (Barendsen et al ., 2015) categorized the task pool for the 2010- 
2014 and found similar dominance of ALG and DR tasks. This paper extends that analy¬ 
sis by looking at individual country selections for each age group and their associated 
performance. 

5.2. Task Difficulty 

When examining the charts in Fig. 7 we can note that, in average, the medium tasks are 
targeted around the 50% success for most age groups and countries. Overall, however, 
there are fewer easy tasks than intended. On the other hand, although the bulk of hard 
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tasks have success rates in expected range, this is usually due to the presence of one or 
two easy tasks that lift their average for almost all age groups. 

Assessing the difficulty of new tasks is a challenging process, and the estimated 
ranks exhibited mixed success. Our observations are consistent with previous studies 
pinpointing a discrepancy between intended and actual difficulty level, regardless of age 
group or country (Bellettini et al ., 2015, van der Vegt, 2013). 

Note that those appreciable performance differences between countries for some of 
the common tasks indicate that an estimation of task difficulty based on CT category 
alone will only be effective for some problems, and there are other factors that caused 
such variable performance. It is hard to speculate what these factors will be, as if differ¬ 
ences are caused by students background knowledge or familiarity with the context of 
the examples, the variations will be more consistent within a given country. 

5.3. Gender 

The Bebras challenge is equally popular among boys and girls in the early years, while 
girls’ participation slowly decreases in relation to that of boys starting around age 12. 
This observation is likely related to the fact that “the gender gap in STEM orientations 
is largely a process that occurs during the high school years” (Legewie and DiPrete, 
2011), but there may be other potential reasons for this change. The overall participa¬ 
tion rate is indeed smaller for Junior and Senior compared to the younger groups. At 
the same time, older pupils are more likely to have developed a personal “identity” 
and wish to choose the activities to get involved in. In addition, the challenge may 
be more commonly delivered in specific STEM courses, where females are underrep¬ 
resented (Justman and Mendez, 2016, Office for Civil Rights & U.S. Department of 
Education, 2012). 

The graph in Fig. 8 shows that, overall, the difference in performance between boys 
and girls was usually small, hence supporting the notion that there is nothing specifi¬ 
cally “male” about computational thinking. On the other hand, our findings are more 
in line with the results of the German study discussed in (Hubwieser et al ., 2016) than 
with the patterns observed by (Dagiene et al ., 2014). In Finland and Lithuania the per¬ 
formance of girls vs. boys is fairly balanced, especially for the first three age groups, 
but overall the situations where girls outperform boys seem to be more of an exception 
than a rule. 

When examining the plotted data, we identified a few tasks on which boys performed 
definitely better than girls in several countries. Those tasks required spatial skills and 
also relied on pupils having an intuitive understanding of binary search, which in its 
turn does have spatial implications to make a mental image of what is going on. Gender 
differences in spatial ability are well known, and have been investigated both in general 
(Linn and Petersen, 1985) and in the specific case of solving Bebras problems (Dagiene 
et al ., 2015). However, even for some of the tasks requiring spatial abilities, the perfor¬ 
mance of boys were unequivocally better only at the Junior and Senior levels. Here is a 
selection of the tasks for which boys performed better than girls: 
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The largest relative difference of performance between boys and girls is attained in 
2014-CH-07 (about 75% for FIN Senior), a task where students are asked to choose 
from a list of four “turtle” programs the only one which does not produce a given draw¬ 
ing - a composition of 5 squares of different size and color. The gender impact is less 
extreme but also large in other countries (40% for SAf Senior and just below 30% for 
SAf Junior and AUS Senior). 

In 2014-FR-03 the student should find a hidden object in a labyrinth by revealing 
parts of the labyrinth by clicking in it as few times as possible. The way in which click¬ 
ing the labyrinth reveals it is shown as one starts clicking. The least number of clicks 
is the result of using a binary search strategy. (Gender differences: SAf Benjamin 55%, 
SAf Cadet: 47%, AUS Mini: 35%, AUS Cadet: 34%, AUS Junior: 42%, LIT Cadet: 
31%, LIT Junior: 60%). 

In 2015-CH-02 the student should navigate to a given room number by going either 
to the left or to the right from the current room number. The goal is to find the correct 
room with as few movements as possible. The best result is achieved by using binary 
search. (Gender differences: FIN Benjamin: 23%, FIN Cadet: 33%, LIT Benjamin: 26%, 
LIT Cadet: 22%, SW Benjamin: 27%). 

In 2014-SI-02 the student is given a two-dimensional simple labyrinth in which a 
robot should find a given object starting from a given position. The student is to find 
out which of four lists of commands written in alien language can take the robot to the 
element. (Gender differences: FIN Senior: 41%, SAf: 25-31% in the two oldest age 
groups). 


6. Conclusions 

This paper has thoroughly analyzed the CT content of Bebras tasks assigned in the 
2014-2015 challenges as well as the related performance of a large multinational cohort 
of students. While trying to answer the third research question introduced in Section 1, 
our study has validated some previous observations based on either smaller or less het¬ 
erogeneous samples: 

(1) Overall, students from different school systems exhibit comparable CT skills. 

(2) A declining performance trend of girls vs. boys can be recognized from primary to 
high school level, with boys outperforming girls in all countries at Senior age. 

(3) Gender differences relative to all age groups were observed for some tasks related 
to spatial skills. 

How to reliably estimate the difficulty of new tasks is still an open issue, for which 
we need to better understand and characterize the range of algorithmic strategies used 
in the challenge, and the factors that caused large task performance variations among 
countries. The CT conceptual content, in particular, does not seem to play a major role 
in assessing how difficult a task is (second research question). However, the fact that 
young students around the world managed to solve many Bebras tasks prior to having 
received any instruction on CT reflects their readiness to learn core CT concepts using 
everyday examples. 
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Our data indicate that the challenge mainly addresses the ALG and DR categories 
(first research question). This is to be expected on the lower age groups, but we should 
provide Junior and Senior students with a variety of tasks that significantly covers other 
CT concepts/skills. Our suggestion to the Bebras community may be to try to take a 
broader perspective on CT in the future. Besides, creating an archive of Bebras tasks 
which can be searched by CT subcategory may improve the use of the challenge as a tool 
to teach CT concepts in the classroom. 
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